Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix message timestamp=0 when archiving MUC legacy messages #4294

Merged
merged 4 commits into from
Jun 5, 2024
Merged

Fix message timestamp=0 when archiving MUC legacy messages #4294

merged 4 commits into from
Jun 5, 2024

Conversation

arcusfelis
Copy link
Contributor

Fix for:

filter_room_packet is called with timestamp=0 (i.e. event_data_timestamp=0 in the logs).
mod_mam_utils:generate_message_id(0).
257

Proposed changes include:

  • set timestamp for messages
  • Add mongoose_mam_id:reset/0 for debugging/testing

Test fails with (without a fix):

{mam_SUITE,'-muc_archive_request_after_mam_id_reset/1-fun-0-',2240}
{test_case_failed,{bad_mam_id,<<"81">>,1,1717450016458975,257}}

Fix for #4269

@arcusfelis
Fix message timestamp=0 when archiving MUC legacy messages
aed71bf 
Add mongoose_mam_id:reset/0 for debugging
@mongoose-im
Copy link
Collaborator

mongoose-im commented Jun 3, 2024 •
edited
Loading

elasticsearch_and_cassandra_26 / elasticsearch_and_cassandra_mnesia / aed71bf
Reports root/ big
OK: 441 / Failed: 0 / User-skipped: 43 / Auto-skipped: 0

small_tests_25 / small_tests / aed71bf
Reports root / small

small_tests_26 / small_tests / aed71bf
Reports root / small

small_tests_26_arm64 / small_tests / aed71bf
Reports root / small

ldap_mnesia_25 / ldap_mnesia / aed71bf
Reports root/ big
OK: 2289 / Failed: 0 / User-skipped: 905 / Auto-skipped: 0

ldap_mnesia_26 / ldap_mnesia / aed71bf
Reports root/ big
OK: 2289 / Failed: 0 / User-skipped: 905 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_25 / pgsql_mnesia / aed71bf
Reports root/ big
OK: 4566 / Failed: 0 / User-skipped: 111 / Auto-skipped: 0

dynamic_domains_mysql_redis_26 / mysql_redis / aed71bf
Reports root/ big
OK: 4533 / Failed: 0 / User-skipped: 144 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_26 / pgsql_mnesia / aed71bf
Reports root/ big
OK: 4566 / Failed: 0 / User-skipped: 111 / Auto-skipped: 0

internal_mnesia_26 / internal_mnesia / aed71bf
Reports root/ big
OK: 2429 / Failed: 0 / User-skipped: 765 / Auto-skipped: 0

dynamic_domains_mssql_mnesia_26 / odbc_mssql_mnesia / aed71bf
Reports root/ big
OK: 4563 / Failed: 0 / User-skipped: 114 / Auto-skipped: 0

pgsql_cets_26 / pgsql_cets / aed71bf
Reports root/ big
OK: 4479 / Failed: 0 / User-skipped: 178 / Auto-skipped: 0

pgsql_mnesia_25 / pgsql_mnesia / aed71bf
Reports root/ big
OK: 4957 / Failed: 0 / User-skipped: 118 / Auto-skipped: 0

pgsql_mnesia_26 / pgsql_mnesia / aed71bf
Reports root/ big
OK: 4957 / Failed: 0 / User-skipped: 118 / Auto-skipped: 0

mysql_redis_26 / mysql_redis / aed71bf
Reports root/ big
OK: 4936 / Failed: 0 / User-skipped: 139 / Auto-skipped: 0

mssql_mnesia_26 / odbc_mssql_mnesia / aed71bf
Reports root/ big
OK: 4954 / Failed: 0 / User-skipped: 121 / Auto-skipped: 0

@codecov Codecov
Copy link

codecov bot commented Jun 3, 2024 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 84.53%. Comparing base (ae06015) to head (5a31967).

Additional details and impacted files 
@@           Coverage Diff           @@
##           master    #4294   +/-   ##
=======================================
  Coverage   84.53%   84.53%           
=======================================
  Files         553      553           
  Lines       33718    33718           
=======================================
+ Hits        28503    28504    +1     
+ Misses       5215     5214    -1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

chrzaszcz
chrzaszcz reviewed Jun 4, 2024
View reviewed changes
Copy link
Member

@chrzaszcz chrzaszcz left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have mixed feelings about this, because it looks like we don't know what causes the issue, so it's fabricated in tests with a NIF that is used only in that test 🙁

big_tests/tests/mam_SUITE.erl Outdated Show resolved Hide resolved
src/muc/mod_muc_room.erl Outdated Show resolved Hide resolved
src/muc/mod_muc_room.erl Outdated Show resolved Hide resolved
big_tests/tests/mam_SUITE.erl Outdated Show resolved Hide resolved
big_tests/tests/mam_SUITE.erl Outdated Show resolved Hide resolved
@mongoose-im
Copy link
Collaborator

mongoose-im commented Jun 4, 2024 •
edited
Loading

elasticsearch_and_cassandra_26 / elasticsearch_and_cassandra_mnesia / ea8c809
Reports root/ big
OK: 441 / Failed: 0 / User-skipped: 43 / Auto-skipped: 0

small_tests_25 / small_tests / ea8c809
Reports root / small

small_tests_26 / small_tests / ea8c809
Reports root / small

small_tests_26_arm64 / small_tests / ea8c809
Reports root / small

ldap_mnesia_25 / ldap_mnesia / ea8c809
Reports root/ big
OK: 2289 / Failed: 0 / User-skipped: 905 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_25 / pgsql_mnesia / ea8c809
Reports root/ big
OK: 4565 / Failed: 1 / User-skipped: 111 / Auto-skipped: 0

mam_SUITE:rdbms_cache_muc_all:muc06:muc_show_x_user_for_your_own_messages_in_anon_rooms 
{error,{test_case_failed,"Respond size is 0, 1 is expected."}}

Report log

ldap_mnesia_26 / ldap_mnesia / ea8c809
Reports root/ big
OK: 2289 / Failed: 0 / User-skipped: 905 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_26 / pgsql_mnesia / ea8c809
Reports root/ big
OK: 4563 / Failed: 3 / User-skipped: 111 / Auto-skipped: 0

mam_SUITE:rdbms_muc_all:muc04:muc_multiple_devices 
{error,
  {function_clause,
    [{exml_query,attr,[undefined,<<"by">>,undefined],[]},
     {mam_SUITE,'-muc_multiple_devices/1-fun-0-',5,
       [{file,"/home/circleci/project/big_tests/tests/mam_SUITE.erl"},
        {line,2291}]},
     {escalus_story,story,4,
       [{file,
          "/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
        {line,72}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1793}]},
     {test_server,run_test_case_eval1,6,
       [{file,"test_server.erl"},{line,1302}]},
     {test_server,run_test_case_eval,9,
       [{file,"test_server.erl"},{line,1234}]}]}}

Report log

mam_SUITE:rdbms_async_cache_muc_all:muc06:muc_sanitize_x_user_in_non_anon_rooms 
{error,{test_case_failed,"Respond size is 0, 1 is expected."}}

Report log

mam_SUITE:rdbms_mnesia_cache_muc_all:muc06:muc_sanitize_x_user_in_non_anon_rooms 
{error,{test_case_failed,"Respond size is 0, 1 is expected."}}

Report log

dynamic_domains_mysql_redis_26 / mysql_redis / ea8c809
Reports root/ big
OK: 4530 / Failed: 3 / User-skipped: 144 / Auto-skipped: 0

mam_SUITE:rdbms_async_cache_muc_all:muc06:muc_archive_request 
{error,
  {function_clause,
    [{exml_query,attr,[undefined,<<"by">>,undefined],[]},
     {mam_SUITE,'-muc_archive_request/1-fun-0-',4,
       [{file,"/home/circleci/project/big_tests/tests/mam_SUITE.erl"},
        {line,2174}]},
     {escalus_story,story,4,
       [{file,
          "/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
        {line,72}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1793}]},
     {test_server,run_test_case_eval1,6,
       [{file,"test_server.erl"},{line,1302}]},
     {test_server,run_test_case_eval,9,
       [{file,"test_server.erl"},{line,1234}]}]}}

Report log

mam_SUITE:rdbms_cache_muc_all:muc06:muc_multiple_devices 
{error,
  {function_clause,
    [{exml_query,attr,[undefined,<<"by">>,undefined],[]},
     {mam_SUITE,'-muc_multiple_devices/1-fun-0-',5,
       [{file,"/home/circleci/project/big_tests/tests/mam_SUITE.erl"},
        {line,2291}]},
     {escalus_story,story,4,
       [{file,
          "/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
        {line,72}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1793}]},
     {test_server,run_test_case_eval1,6,
       [{file,"test_server.erl"},{line,1302}]},
     {test_server,run_test_case_eval,9,
       [{file,"test_server.erl"},{line,1234}]}]}}

Report log

mam_SUITE:rdbms_cache_muc_all:muc06:muc_delete_x_user_in_anon_rooms 
{error,{test_case_failed,"Respond size is 0, 1 is expected."}}

Report log

internal_mnesia_26 / internal_mnesia / ea8c809
Reports root/ big
OK: 2429 / Failed: 0 / User-skipped: 765 / Auto-skipped: 0

pgsql_cets_26 / pgsql_cets / ea8c809
Reports root/ big
OK: 4477 / Failed: 2 / User-skipped: 178 / Auto-skipped: 0

mam_SUITE:rdbms_muc_all:muc04:muc_sanitize_x_user_in_non_anon_rooms 
{error,{test_case_failed,"Respond size is 0, 1 is expected."}}

Report log

mam_SUITE:rdbms_async_pool_muc_all:muc04:muc_text_search_request 
{error,{test_case_failed,"Respond size is 2, 3 is expected."}}

Report log

dynamic_domains_mssql_mnesia_26 / odbc_mssql_mnesia / ea8c809
Reports root/ big
OK: 4561 / Failed: 2 / User-skipped: 114 / Auto-skipped: 0

mam_SUITE:rdbms_async_pool_muc_all:muc04:muc_archive_request 
{error,
  {function_clause,
    [{exml_query,attr,[undefined,<<"by">>,undefined],[]},
     {mam_SUITE,'-muc_archive_request/1-fun-0-',4,
       [{file,"/home/circleci/project/big_tests/tests/mam_SUITE.erl"},
        {line,2174}]},
     {escalus_story,story,4,
       [{file,
          "/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
        {line,72}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1793}]},
     {test_server,run_test_case_eval1,6,
       [{file,"test_server.erl"},{line,1302}]},
     {test_server,run_test_case_eval,9,
       [{file,"test_server.erl"},{line,1234}]}]}}

Report log

mam_SUITE:rdbms_mnesia_muc_all:muc06:muc_message_with_stanzaid 
{error,
  {function_clause,
    [{exml_query,attr,[undefined,<<"xmlns">>,undefined],[]},
     {mam_SUITE,'-muc_message_with_stanzaid/1-fun-0-',3,
       [{file,"/home/circleci/project/big_tests/tests/mam_SUITE.erl"},
        {line,2084}]},
     {escalus_story,story,4,
       [{file,
          "/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
        {line,72}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1793}]},
     {test_server,run_test_case_eval1,6,
       [{file,"test_server.erl"},{line,1302}]},
     {test_server,run_test_case_eval,9,
       [{file,"test_server.erl"},{line,1234}]}]}}

Report log

pgsql_mnesia_25 / pgsql_mnesia / ea8c809
Reports root/ big
OK: 4956 / Failed: 1 / User-skipped: 118 / Auto-skipped: 0

mam_SUITE:rdbms_muc_all:muc06:retract_muc_message 
{error,{test_case_failed,"Respond size is 1, 2 is expected."}}

Report log

mssql_mnesia_26 / odbc_mssql_mnesia / ea8c809
Reports root/ big
OK: 4953 / Failed: 1 / User-skipped: 121 / Auto-skipped: 0

mam_SUITE:rdbms_cache_muc_all:muc04:muc_text_search_request 
{error,{test_case_failed,"Respond size is 2, 3 is expected."}}

Report log

pgsql_mnesia_26 / pgsql_mnesia / ea8c809
Reports root/ big
OK: 4953 / Failed: 4 / User-skipped: 118 / Auto-skipped: 0

mam_SUITE:rdbms_mnesia_muc_all:muc06:muc_delete_x_user_in_anon_rooms 
{error,{test_case_failed,"Respond size is 0, 1 is expected."}}

Report log

mam_SUITE:rdbms_cache_muc_all:muc06:muc_delete_x_user_in_anon_rooms 
{error,{test_case_failed,"Respond size is 0, 1 is expected."}}

Report log

mam_SUITE:rdbms_mnesia_cache_muc_all:muc04:muc_text_search_request 
{error,{test_case_failed,"Respond size is 2, 3 is expected."}}

Report log

mam_SUITE:rdbms_mnesia_cache_muc_all:muc06:retract_muc_message 
{error,{test_case_failed,"Respond size is 1, 2 is expected."}}

Report log

mysql_redis_26 / mysql_redis / ea8c809
Reports root/ big
OK: 4935 / Failed: 1 / User-skipped: 139 / Auto-skipped: 0

mam_SUITE:rdbms_mnesia_muc_all:muc04:muc_archive_request 
{error,
  {function_clause,
    [{exml_query,attr,[undefined,<<"by">>,undefined],[]},
     {mam_SUITE,'-muc_archive_request/1-fun-0-',4,
       [{file,"/home/circleci/project/big_tests/tests/mam_SUITE.erl"},
        {line,2174}]},
     {escalus_story,story,4,
       [{file,
          "/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
        {line,72}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1793}]},
     {test_server,run_test_case_eval1,6,
       [{file,"test_server.erl"},{line,1302}]},
     {test_server,run_test_case_eval,9,
       [{file,"test_server.erl"},{line,1234}]}]}}

Report log

@mongoose-im
Copy link
Collaborator

mongoose-im commented Jun 4, 2024 •
edited
Loading

elasticsearch_and_cassandra_26 / elasticsearch_and_cassandra_mnesia / e9a08b5
Reports root/ big
OK: 439 / Failed: 0 / User-skipped: 43 / Auto-skipped: 0

small_tests_25 / small_tests / e9a08b5
Reports root / small

small_tests_26 / small_tests / e9a08b5
Reports root / small

small_tests_26_arm64 / small_tests / e9a08b5
Reports root / small

ldap_mnesia_25 / ldap_mnesia / e9a08b5
Reports root/ big
OK: 2289 / Failed: 0 / User-skipped: 905 / Auto-skipped: 0

dynamic_domains_mysql_redis_26 / mysql_redis / e9a08b5
Reports root/ big
OK: 4526 / Failed: 0 / User-skipped: 144 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_25 / pgsql_mnesia / e9a08b5
Reports root/ big
OK: 4559 / Failed: 0 / User-skipped: 111 / Auto-skipped: 0

ldap_mnesia_26 / ldap_mnesia / e9a08b5
Reports root/ big
OK: 2289 / Failed: 0 / User-skipped: 905 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_26 / pgsql_mnesia / e9a08b5
Reports root/ big
OK: 4559 / Failed: 0 / User-skipped: 111 / Auto-skipped: 0

internal_mnesia_26 / internal_mnesia / e9a08b5
Reports root/ big
OK: 2429 / Failed: 0 / User-skipped: 765 / Auto-skipped: 0

pgsql_cets_26 / pgsql_cets / e9a08b5
Reports root/ big
OK: 4473 / Failed: 1 / User-skipped: 178 / Auto-skipped: 0

service_domain_db_SUITE:db:plain_db_sequential:db_out_of_sync_restarts_service 
{error,
  {{badmatch,false},
   [{service_domain_db_SUITE,db_out_of_sync_restarts_service,1,
      [{file,
         "/home/circleci/project/big_tests/tests/service_domain_db_SUITE.erl"},
       {line,711}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1793}]},
    {test_server,run_test_case_eval1,6,
      [{file,"test_server.erl"},{line,1302}]},
    {test_server,run_test_case_eval,9,
      [{file,"test_server.erl"},{line,1234}]}]}}

Report log

dynamic_domains_mssql_mnesia_26 / odbc_mssql_mnesia / e9a08b5
Reports root/ big
OK: 4556 / Failed: 0 / User-skipped: 114 / Auto-skipped: 0

mysql_redis_26 / mysql_redis / e9a08b5
Reports root/ big
OK: 4929 / Failed: 0 / User-skipped: 139 / Auto-skipped: 0

pgsql_mnesia_25 / pgsql_mnesia / e9a08b5
Reports root/ big
OK: 4950 / Failed: 0 / User-skipped: 118 / Auto-skipped: 0

pgsql_mnesia_26 / pgsql_mnesia / e9a08b5
Reports root/ big
OK: 4950 / Failed: 0 / User-skipped: 118 / Auto-skipped: 0

mssql_mnesia_26 / odbc_mssql_mnesia / e9a08b5
Reports root/ big
OK: 4947 / Failed: 0 / User-skipped: 121 / Auto-skipped: 0

small_tests_26 / small_tests / e9a08b5
Reports root / small

pgsql_cets_26 / pgsql_cets / e9a08b5
Reports root/ big
OK: 4474 / Failed: 0 / User-skipped: 178 / Auto-skipped: 0

chrzaszcz
chrzaszcz approved these changes Jun 5, 2024
View reviewed changes
Copy link
Member

@chrzaszcz chrzaszcz left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good now, thanks for the update 👍

In my previous review, I misunderstood how this was supposed to work.

big_tests/tests/mam_SUITE.erl Outdated Show resolved Hide resolved
@chrzaszcz chrzaszcz closed this Jun 5, 2024
@chrzaszcz chrzaszcz reopened this Jun 5, 2024
@mongoose-im
Copy link
Collaborator

mongoose-im commented Jun 5, 2024 •
edited
Loading

elasticsearch_and_cassandra_26 / elasticsearch_and_cassandra_mnesia / 5a31967
Reports root/ big
OK: 439 / Failed: 0 / User-skipped: 43 / Auto-skipped: 0

small_tests_25 / small_tests / 5a31967
Reports root / small

small_tests_26 / small_tests / 5a31967
Reports root / small

small_tests_26_arm64 / small_tests / 5a31967
Reports root / small

ldap_mnesia_26 / ldap_mnesia / 5a31967
Reports root/ big
OK: 2289 / Failed: 0 / User-skipped: 905 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_25 / pgsql_mnesia / 5a31967
Reports root/ big
OK: 4559 / Failed: 0 / User-skipped: 111 / Auto-skipped: 0

ldap_mnesia_25 / ldap_mnesia / 5a31967
Reports root/ big
OK: 2289 / Failed: 0 / User-skipped: 905 / Auto-skipped: 0

dynamic_domains_mysql_redis_26 / mysql_redis / 5a31967
Reports root/ big
OK: 4526 / Failed: 0 / User-skipped: 144 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_26 / pgsql_mnesia / 5a31967
Reports root/ big
OK: 4559 / Failed: 0 / User-skipped: 111 / Auto-skipped: 0

dynamic_domains_mssql_mnesia_26 / odbc_mssql_mnesia / 5a31967
Reports root/ big
OK: 4556 / Failed: 0 / User-skipped: 114 / Auto-skipped: 0

internal_mnesia_26 / internal_mnesia / 5a31967
Reports root/ big
OK: 2429 / Failed: 0 / User-skipped: 765 / Auto-skipped: 0

pgsql_mnesia_25 / pgsql_mnesia / 5a31967
Reports root/ big
OK: 4950 / Failed: 0 / User-skipped: 118 / Auto-skipped: 0

pgsql_cets_26 / pgsql_cets / 5a31967
Reports root/ big
OK: 4474 / Failed: 0 / User-skipped: 178 / Auto-skipped: 0

mysql_redis_26 / mysql_redis / 5a31967
Reports root/ big
OK: 4929 / Failed: 0 / User-skipped: 139 / Auto-skipped: 0

pgsql_mnesia_26 / pgsql_mnesia / 5a31967
Reports root/ big
OK: 4950 / Failed: 0 / User-skipped: 118 / Auto-skipped: 0

mssql_mnesia_26 / odbc_mssql_mnesia / 5a31967
Reports root/ big
OK: 4947 / Failed: 0 / User-skipped: 121 / Auto-skipped: 0

@chrzaszcz chrzaszcz merged commit 83cabbf into master Jun 5, 2024
5 of 6 checks passed
@chrzaszcz chrzaszcz deleted the mu-mam-muc-message-timestamp-bug branch June 5, 2024 14:05
@jacekwegr jacekwegr added this to the 6.3.0 milestone Oct 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@chrzaszcz chrzaszcz chrzaszcz approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
6.3.0
Development

Successfully merging this pull request may close these issues.
4 participants
@arcusfelis @mongoose-im @chrzaszcz @jacekwegr